Skip to content

fix(ingestion): stop duplicate instants reaching one INSERT - #862

Merged
jirhiker merged 1 commit into
stagingfrom
fix/duplicate-rows-in-batch
Aug 19, 2026
Merged

fix(ingestion): stop duplicate instants reaching one INSERT#862
jirhiker merged 1 commit into
stagingfrom
fix/duplicate-rows-in-batch

Conversation

@jirhiker

Copy link
Copy Markdown
Member

The first live materialization failed. Both raw assets succeeded;
san_acacia_observations failed three times (initial + 2 retries) and took the
run down.

ON CONFLICT DO UPDATE command cannot affect row a second time
HINT: Ensure that no rows proposed for insertion within the same command
      have duplicate constrained values.

Nothing was written — the transaction rolled back each time, so staging is
unchanged and holds only the 542,161 pre-existing AMPAPI rows.

Two causes, both mine

Overlapping windows. Diver-HUB ranges are inclusive at both ends — "from
start time up to and including end time" — but iter_windows made adjacent
windows share a boundary: [0, span] then [span, 2·span]. A reading logged
exactly on the boundary came back in both.

No deduplication in the loader. That pair then reached Postgres inside one
INSERT, which the constraint from #847 correctly refuses.

Fixes

Windows now leave exactly one second between them. Timestamps are
second-resolution, so nothing falls in the gap — there is a test for that.

The loader deduplicates within a batch, keeping the last occurrence, which
matches the upsert's own rule that a later value wins. That guard holds whatever
the source does, including a vendor logging one instant twice — so this does not
depend on the window fix being perfect.

Why the existing tests missed it

test_loading_the_same_window_twice_does_not_duplicate loads the same window in
two separate statements, which Postgres allows. The failure needs duplicates
in one statement. The new test does exactly that and asserts 6 records
become 3 rows with the later value winning.

Diagnosis note

RetryPolicy obscured this: the surfaced error was "Exceeded max_retries of 2",
and the real cause was two levels down under error.cause. Worth knowing when
reading a failed run.

125 ingestion tests plus 11 loader tests pass.

🤖 Generated with Claude Code

The first live materialization failed:

  ON CONFLICT DO UPDATE command cannot affect row a second time
  HINT: Ensure that no rows proposed for insertion within the same command
        have duplicate constrained values.

Two causes, both here. Diver-HUB ranges are inclusive at both ends -- "up to
and including end time" -- while iter_windows made adjacent windows share a
boundary, so a reading logged exactly on it came back in both. And the loader
never deduplicated, so that pair reached Postgres in one statement.

Windows now leave exactly one second between them. Timestamps are
second-resolution, so nothing falls in the gap.

The loader also deduplicates within a batch, keeping the last occurrence --
which matches the upsert's own rule that a later value wins. That guard holds
whatever the source does, including a vendor logging one instant twice.

The existing idempotency test could not have caught this: it loads the same
window in two separate statements, which Postgres allows. The new test puts the
duplicates in one batch, which is what actually happened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Coverage

78.59% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
automated_ingestion/ocotillo/loader.py 60 14 77% 60, 142, 196-225
automated_ingestion/shared/windows.py 32 2 94% 54, 74
TOTAL 92 16 83%

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
ocotillo-automated-ingestion View in Cloud Aug 19, 2026 at 08:21 PM (UTC)

@jirhiker
jirhiker merged commit ef38ef2 into staging Aug 19, 2026
10 checks passed
@jirhiker
jirhiker deleted the fix/duplicate-rows-in-batch branch August 19, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant